From large business transaction to personal financial management, almost every aspect of our daily life depends on the secure operation of computer networks. People have invested heavily in protecting computer networks from attacks.
The first line of defense against computer network attacks is the denial of access through passwords and firewalls at the node level of a computer network. A limitation of this approach is that it focuses only on certain localized sections of a computer network. However, in many circumstances, a serious attack may comprise multiple steps. Capturing only some of them is not enough for detecting and thwarting the attack.
In order to overcome the deficiency discussed above, multiple security devices such as intrusion detection sensors (IDS) are deployed at different sections of a computer network to detect multiple security-related events simultaneously. As shown in FIG. 1, security devices are attached to routers, firewalls, switches and hosts, etc. Each security device is configured such that whenever it detects a suspicious event, e.g., an IP packet, it sends an event message to a network security monitor. The network security monitor is responsible for correlating diverse events from different parts of the network and providing insights into higher-level attack scenarios.
For various reasons, the data and traffic volume associated with security events have increased dramatically over time. For example, a large number of intrusions can be automated and launched simultaneously from geographically dispersed locations, network link speed is increasing, security devices are becoming faster and generating more data, and single attack can cause multiple events to be generated from various security devices that lie on different network topological paths of that attack. Databases have been used to store this large volume of event messages, and different queries have been designed to correlate multiple event messages in order to detect a high-level attack. A problem with this methodology is that while it performs well as an offline process, it does not scale well to handle a high volume of incoming events in real time. This is because event correlation queries can take substantial time to execute, and such queries have to be repeated either for every received event message, or at periodic intervals in order to provide real time performance. Handling event messages in a real-time fashion is critical for a security event detection system, because only real-time intrusion detected enables mitigation actions to be taken early, before substantial damage is incurred.
Therefore, it would be highly desirable to have a method and apparatus that can correlate event messages in real time as event messages arrive at a network security monitor, and to thereby detect in real time security attacks involving multiple events, even when the attacks include packets sent by multiple distinct sources.